Skip Headers

Oracle® Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide 10g Release 2 (10.2) Part Number B14197-03 |
Home | Book List | Contents | Index | Master Index | Feedback

Previous | Next
| View PDF

# 10 Adding and Deleting Nodes and Instances on UNIX-Based Systems

This chapter describes how to add and delete nodes and instances in Oracle Real Application Clusters (RAC) databases on UNIX-based systems. The preferred method to add nodes and instances to RAC databases is to use the Oracle cloning procedures that are described in the Oracle Universal Installer and OPatch User's Guide. You can also use Enterprise Manager Grid Control to perform cloning operations. Cloning enables you to copy images of Oracle Clusterware and RAC software onto the other nodes that have identical hardware and software. If you do not want to use the cloning process, then you can use the manual procedures that are described in this chapter to add nodes and instances. The topics in this chapter are:

- Cloning Oracle Clusterware and RAC Software in Grid Environments

- Quick-Start Node and Instance Addition and Deletion Procedures

- Detailed Node and Instance Addition and Deletion Procedures

- Step 3: ASM Instance Clean-Up Procedures for Node Deletion

  Note: For all of the add node and delete node procedures for UNIX-based systems, temporary directories such as `/tmp`, `$TEMP`, or `$TMP`, should not be shared directories. If your temporary directories are shared, then set your temporary environment variable, such as `$TEMP`, to a non-shared location on a local node. In addition, use a directory that exists on all of the nodes.

  Note: The entries for `CRS_home` and `Oracle_home` as used in this chapter refer to substitutes for either environment variables or full path names for Oracle Clusterware and Oracle home with RAC respectively.

## Cloning Oracle Clusterware and RAC Software in Grid Environments

The preferred method for extending your RAC environment is to use the Oracle cloning procedures in the Oracle Universal Installer and OPatch User's Guide. Only use the procedures in this chapter if you choose not to use Oracle cloning.

See Also: "Cloning Oracle Clusterware and RAC Software in Grid Environments" for a summary about Oracle cloning

## Quick-Start Node and Instance Addition and Deletion Procedures

This section explains node and instance addition and deletion for UNIX-based systems using procedures that are presented in a quick-start format. For node addition, ensure that you install the required operating system patches and updates on the new nodes. Then configure the new nodes to be part of your cluster at the network level. Extend the Oracle Clusterware home from an existing Oracle Clusterware home to the new nodes and then extend the Oracle database software with RAC components to the new nodes. Finally, make the new nodes members of the existing RAC database. This section includes the following topics:

- Adding an Oracle Clusterware Home to a New Node

- Adding an Oracle Home with RAC to a New Node

- Deleting an Oracle Home with RAC from an Existing Node

- Deleting an Oracle Clusterware Home from an Existing Node

Note: If you are using Oracle Clusterware without vendor clusterware, then you can add and delete nodes without stopping the existing nodes. If you are using Oracle Clusterware with vendor clusterware, then you can add nodes on some UNIX-based systems without stopping the existing nodes if your clusterware supports this. Refer to your vendor-specific clusterware documentation for more information.

Note: For all of the procedures in this chapter, it is very important that you perform each step in the order shown.

See Also: Your Oracle Real Application Clusters platform-specific installation and configuration guide for procedures about using the Database Configuration Assistant (DBCA) to create and delete RAC databases

### Adding an Oracle Clusterware Home to a New Node

Use one of the procedures described in this section to use OUI to add an Oracle Clusterware home to a node that is going to be part of your RAC cluster. These procedures assume that you are adding `node2` and that you have already successfully installed Oracle Clusterware on `node1` in a non-shared home, where `CRS_home` represents the successfully installed home. This section describes the following procedures:

- Adding an Oracle Clusterware Home to a New Node Using OUI in Interactive Mode

- Adding an Oracle Clusterware Home to a New Node Using OUI in Silent Mode

  Note: When you extend a RAC database, you must first extend the Oracle Clusterware home to the new nodes and then extend the Oracle home. In other words, you extend the software onto the new nodes in the same order as you installed the clusterware and Oracle database software components on the existing nodes.

  The following sections are based on the addition of a node named `node2` to an existing cluster that has a node named `node1`.

#### Adding an Oracle Clusterware Home to a New Node Using OUI in Interactive Mode

1. Ensure that you have successfully installed Oracle Clusterware on at least one node in your cluster environment. To use these procedures as shown, your `$CRS_HOME` environment variable must identify your successfully installed Oracle Clusterware home.

2. Go to `CRS_home``/oui/bin` and run the `addNode.sh` script.

3. The Oracle Universal Installer (OUI) displays the Node Selection Page on which you should select the node or nodes that you want to add and click Next.

4. Verify the entries that OUI displays on the Summary Page and click Next.

5. Run the `rootaddNode.sh` script from the `CRS_home/install/` directory on the node from which you are running OUI.

6. Run the `orainstRoot.sh` script on the new node if OUI prompts you to do so.

7. Run the `root.sh` script on the new node from `CRS_home` to start Oracle Clusterware on the new node.

8. Obtain the remote port identifier, which you need to know for the next step, by running the following command on the existing node from the `CRS_home``/opmn/conf` directory:

       cat ons.config

9. From the `CRS_home``/bin` directory on an existing node, run the Oracle Notification Service (RACGONS) utility as in the following example where `remote_port` is the port number from the previous step and `node2` is the name of the node that you are adding:

       ./racgons add_config node2:remote_port

#### Adding an Oracle Clusterware Home to a New Node Using OUI in Silent Mode

1. Ensure that you have successfully installed Oracle Clusterware on at least one node in your cluster environment. To use these procedures as shown, your `$CRS_HOME` environment variable must identify your successfully installed Oracle Clusterware home.

2. Go to `CRS_home``/oui/bin` and run the `addNode.sh` script using the following syntax where `node2` is the name of the new node that you are adding, `node2-priv` is the private node name for the new node, and `node2-vip` is the VIP name for the new node:

       ./addNode.sh –silent "CLUSTER_NEW_NODES={node2}"
       "CLUSTER_NEW_PRIVATE_NODE_NAMES={node2-priv}"
       "CLUSTER_NEW_VIRTUAL_HOSTNAMES={node2-vip}"

3. Perform OUI-related steps 5 through 9 from the previous section about using OUI interactively under the heading "Adding an Oracle Clusterware Home to a New Node Using OUI in Interactive Mode".

### Adding an Oracle Home with RAC to a New Node

Use one of the following procedures to add an Oracle home to a new node using OUI. These procedures assume that you want to extend an existing Oracle home with RAC on `node1` to `node2`. `Node2` should already be a member node of the cluster to which `node1` belongs. The contents of these procedures assume that you have successfully installed RAC on `node1` in a non-shared home and that `Oracle_home` represents the successfully installed Oracle home. This section describes the following procedures:

- Adding an Oracle Home with RAC to a New Node Using OUI in Interactive Mode

- Adding an Oracle Home with RAC to a New Node Using OUI in Silent Mode

#### Adding an Oracle Home with RAC to a New Node Using OUI in Interactive Mode

1. Ensure that you have successfully installed Oracle with the RAC software on at least one node in your cluster environment. To use these procedures as shown, your `$ORACLE_HOME` environment variable must identify your successfully installed Oracle home.

2. Go to `Oracle_home``/oui/bin` and run the `addNode.sh` script.

3. When OUI displays the Node Selection Page, select the node or nodes to be added and click Next.

4. Verify the entries that OUI displays on the Summary Page and click Next.

5. Run the `root.sh` script on the new node from `Oracle_home` when OUI prompts you to do so.

6. On the new node, run the Oracle Net Configuration Assistant (NETCA) to add a Listener.

7. Use the Enterprise Manager or DBCA to add an instance as described under the heading "Step 5: Adding Database Instances to New Nodes".

See Also: Oracle Database Net Services Administrator's Guide for more information about NETCA

#### Adding an Oracle Home with RAC to a New Node Using OUI in Silent Mode

1. Ensure that you have successfully installed Oracle with the RAC software on at least one node in your cluster environment. To use these procedures, your `$ORACLE_HOME` environment variable must identify your successfully installed Oracle home and the node to be added is named `node2`.

2. Go to `Oracle_home``/oui/bin` and run the `addNode.sh` script using the following syntax:

       ./addNode.sh -silent "CLUSTER_NEW_NODES={node2}"

3. Perform steps 5 through 7 from the previous section about using OUI interactively under the heading "Adding an Oracle Home with RAC to a New Node Using OUI in Interactive Mode".

### Deleting an Oracle Home with RAC from an Existing Node

Use the following procedures to delete nodes at the database Oracle home layer on Windows-based systems. These procedures assume that a RAC database home is installed on both `node1` and `node2` and that you want to delete the database home on node2. Also assume that `Oracle_home` represents the location of this home on the nodes. The procedures for deleting an Oracle home assume that you have successfully installed RAC on the node from which you want to delete the Oracle home. You can use either of the following procedures to delete an Oracle home from a node:

- Deleting an Oracle Home with RAC Using OUI in Interactive Mode

- Deleting an Oracle Home with RAC Using OUI in Silent Mode

  Note: When you remove Oracle Clusterware and RAC database software, you must first remove the Oracle database software and then remove Oracle Clusterware from the nodes that you are deleting. In other words, you remove the software components from the nodes that you are deleting in the reverse order that you originally installed the clusterware and Oracle database software components.

Note: For all of the procedures in this chapter, it is very important that you perform each step in the order shown.

#### Deleting an Oracle Home with RAC Using OUI in Interactive Mode

1. From a node that is to remain a part of your RAC database environment, use Enterprise Manager or DBCA to remove any database instances from the node that you want to delete if instances exist on that node. To do this, refer to the procedure titled "Step 1: Deleting Instances from Real Application Clusters Databases".

2. If you use ASM and this is the home from which the ASM instance runs, then perform the ASM clean up procedures as described under "Step 3: ASM Instance Clean-Up Procedures for Node Deletion".

3. On the node to be deleted, ensure that the `$ORACLE_HOME` environment variable is set to the Oracle home then run the following command from `Oracle_home``/oui/bin` where `node_name` is the name of the node to be deleted:

       ./runInstaller -updateNodeList ORACLE_HOME=Oracle_home
       "CLUSTER_NODES={node_name}" –local

4. Depending on whether you have a shared or non-shared Oracle home, complete one of the following two procedures:

   - For a shared home, run the following command on each of the nodes that are to be deleted, using the shared home location for the `$ORACLE_HOME` environment variable on each node:

         ./runInstaller -detachHome -local ORACLE_HOME=Oracle_home

   - For a non-shared home, on each node that you are deleting, run OUI from the home and deinstall this home. Make sure that you choose the home to be removed and not just the products under that home.

5. On any node other than the node that you are deleting, make sure that the `$ORACLE_HOME` environment variable is defined for the Oracle home that you are deleting. Run the following command from `Oracle_home``/oui/bin` where `remaining_nodes_list` is a comma-delimited list of the nodes that are going to remain as part of your RAC database:

       ./runInstaller -updateNodeList ORACLE_HOME=Oracle_home
       "CLUSTER_NODES={remaining_nodes_list}"

See Also: Oracle Database Net Services Administrator's Guide for more information about NETCA

#### Deleting an Oracle Home with RAC Using OUI in Silent Mode

1. Perform steps 1 through 4 from the previous section about using OUI interactively under the heading "Deleting an Oracle Home with RAC Using OUI in Interactive Mode".

2. Depending on whether you have a shared or non-shared Oracle home, complete one of the following two procedures:

   - For a shared home, ensure that the `$ORACLE_HOME` environment variable is defined for the shared home location and run the following command on each of the nodes that are to be deleted:

         ./runInstaller -detachHome -local ORACLE_HOME=Oracle_home

   - For a non-shared home, on each node that you are deleting, ensure that the `$ORACLE_HOME` environment variable is defined for the home that you are deleting and perform the following step:

         ./runInstaller -deinstall -silent "REMOVE_HOMES={Oracle_home}"

3. Perform Step 5 from the procedure, "Deleting an Oracle Home with RAC Using OUI in Interactive Mode".

### Deleting an Oracle Clusterware Home from an Existing Node

The procedures for deleting an Oracle Clusterware home assume that you have successfully installed the clusterware on the node from which you want to delete the Oracle Clusterware home. You can use either of the following procedures to delete an Oracle Clusterware home from a node:

- Deleting an Oracle Clusterware Home Using OUI in Interactive Mode

- Deleting an Oracle Clusterware Home Using OUI in Silent Mode

#### Deleting an Oracle Clusterware Home Using OUI in Interactive Mode

1. Perform the delete node operation for database homes as described in the section titled "Deleting an Oracle Home with RAC Using OUI in Interactive Mode" or use the procedure, "Deleting an Oracle Home with RAC Using OUI in Silent Mode", and ensure that the `$CRS_HOME` environment variable is defined to identify the appropriate Oracle Clusterware home on each node.

2. If you ran the Oracle Interface Configuration Tool (OIFCFG) with the `-global` flag during the installation, then skip this step. Otherwise, from a node that is going to remain in your cluster, from the `CRS_home``/bin` directory, run the following command where `node2` is the name of the node that you are deleting:

       ./oifcfg delif –node node2

3. Obtain the remote port number, which you will use in the next step, using the following command from the `CRS_home``/opmn/conf` directory:

       cat ons.config

4. On the node to be deleted, run `rootdelete.sh` as the `root` user from the `CRS_home/install` directory. If you are deleting more than one node, then perform this step on all of the other nodes that you are deleting.

5. From any node that you are not deleting, run the following command from the `CRS_home/install` directory as the `root` user where `node2,node2-number` represents the node and the node number that you want to delete:

       ./rootdeletenode.sh node2,node2-number

   If necessary, identify the node number using the following command on the node that you are deleting:

       CRS_home/bin/olsnodes -n

6. On the node or nodes to be deleted, run the following command from the `CRS_home``/oui/bin` directory where `node_to_be_deleted` is the name of the node that you are deleting:

       ./runInstaller -updateNodeList ORACLE_HOME=CRS_home
       "CLUSTER_NODES={node_to_be_deleted}"
       CRS=TRUE -local

7. On the node that you are deleting, run OUI using the `runInstaller` command from the `CRS_home``/oui/bin` directory. Depending on whether you have a shared or non-shared Oracle home, complete one of the following procedures:

   - If you have a shared home, then on any node other than the node to be deleted, run the following command from the `CRS_home``/oui/bin` directory:

         ./runInstaller -detachHome -silent ORACLE_HOME=CRS_home

   - If you have a non-shared home, then de-install Oracle Clusterware using OUI by running the `runInstaller` command from the `CRS_home``/oui/bin` directory.

8. On any node other than the node that you are deleting, run the following command from the `CRS_home``/oui/bin` directory where `remaining_nodes_list` is a comma-delimited list of the nodes that are going to remain part of your RAC database:

       ./runInstaller -updateNodeList ORACLE_HOME=CRS_home
       "CLUSTER_NODES={remaining_nodes_list}"
       CRS=TRUE

#### Deleting an Oracle Clusterware Home Using OUI in Silent Mode

1. Perform steps 1 through 7 from the previous section about using OUI interactively under the heading "Deleting an Oracle Clusterware Home Using OUI in Interactive Mode". Also ensure that the `$CRS_HOME` environment variable is defined to identify the appropriate Oracle Clusterware home on each node.

2. Depending on whether you have a shared or non-shared Oracle Clusterware home, complete one of the following two procedures:

   - For shared homes, do not perform a de-install operation. Instead, perform a detach home operation on the node that you are deleting. To do this, run the following command from `CRS_home``/oui/bin`:

         ./runInstaller -detachHome ORACLE_HOME=CRS_home

   - For a non-shared home, de-install the Oracle Clusterware home from the node that you are deleting using OUI as follows by running the following command from the `Oracle_home``/oui/bin` directory, where `CRS_home` is the name defined for the Oracle Clusterware home:

         ./runInstaller -deinstall –silent "REMOVE_HOMES={CRS_home}"

3. Perform step 9 from the previous section about using OUI interactively under the heading "Deleting an Oracle Clusterware Home Using OUI in Interactive Mode".

Note: Oracle recommends that you back up your voting disk and OCR files after you complete the node deletion process.

## Detailed Node and Instance Addition and Deletion Procedures

This section provides detailed procedures for adding and deleting nodes to clusters. The details of these steps appear in the following sections:

- Overview of Node Addition Procedures

- Step 1: Connecting New Nodes to the Cluster

- Step 2: Extending Clusterware and Oracle Software to New Nodes

- Step 3: Preparing Storage on New Nodes

- Step 4: Adding Nodes at the Oracle RAC Database Layer

- Step 5: Adding Database Instances to New Nodes

- Adding Nodes that Already Have Clusterware and Oracle Software to a Cluster

- Overview of Node Deletion Procedures

- Step 1: Deleting Instances from Real Application Clusters Databases

- Step 2: Deleting Nodes from Real Application Clusters Databases

- Step 3: ASM Instance Clean-Up Procedures for Node Deletion

## Overview of Node Addition Procedures

This section explains how to add nodes to clusters using detailed manual procedures. Do this by setting up the new nodes to be part of your cluster at the network level. Then extend the Oracle Clusterware home from an existing Oracle Clusterware home to the new nodes and then extend the Oracle database software with RAC components to the new nodes. Finally, make the new nodes members of the existing RAC database.

Note: You can add nodes on some UNIX-based platforms without stopping existing nodes if your clusterware supports this. Refer to your vendor-specific clusterware documentation for more information.

If the nodes that you are adding to your cluster do not have clusterware or Oracle software installed on them and if you are not using Oracle cloning, then you must complete the following five steps to add nodes. The procedures in these steps assume that you already have an operative UNIX-based RAC environment. The details of these steps appear in the following sections.

- Step 1: Connecting New Nodes to the Cluster

- Step 2: Extending Clusterware and Oracle Software to New Nodes

- Step 3: Preparing Storage on New Nodes

- Step 4: Adding Nodes at the Oracle RAC Database Layer

- Step 5: Adding Database Instances to New Nodes

To add a node to your cluster when the node is already configured with clusterware and Oracle software, follow the procedure described in "Adding Nodes that Already Have Clusterware and Oracle Software to a Cluster".

See Also: Your platform-specific Real Application Clusters installation and configuration guide for procedures about using DBCA to create and delete RAC databases

## Step 1: Connecting New Nodes to the Cluster

Complete the following procedures to connect the new nodes to the cluster and to prepare them to support your RAC database:

- Making Physical Connections

- Installing the Operating System

- Creating Oracle Users

- Verifying the Installation with the Cluster Verification Utility

- Checking the Installation

### Making Physical Connections

Connect the new nodes' hardware to the network infrastructure of your cluster. This includes establishing electrical connections, configuring network interconnects, configuring shared disk subsystem connections, and so on. Refer to your hardware vendor documentation for details about this step.

### Installing the Operating System

Install a cloned image of the operating system that matches the operating system on the other nodes in your cluster. This includes installing required service patches and drivers. Refer to your hardware vendor documentation for details about this process.

### Creating Oracle Users

As `root` user, create the Oracle users and groups using the same user ID and group ID as on the existing nodes.

### Verifying the Installation with the Cluster Verification Utility

Verify your installation using the Cluster Verification Utility (CVU) as in the following steps:

1. From the directory `CRS_home``/bin` on the existing nodes, run the CVU command to verify your installation at the post hardware installation stage as shown in the following example, where `node_list` is a comma-delimited list of nodes you want in your cluster:

       cluvfy stage -post hwos -n node_list|all [-verbose]

   This command causes CVU to verify your hardware and operating system environment at the post-hardware setup stage. After you have configured the hardware and operating systems on the new nodes, you can use this command to verify node reachability, for example, to all of the nodes from the local node. You can also use this command to verify user equivalence to all given nodes the local node, node connectivity among all of the given nodes, accessibility to shared storage from all of the given nodes, and so on.

   Note: You can only use the `all` option with the `-n` argument if you have set the `CV_NODELIST` variable to represent the list of nodes on which you want to perform the CVU operation.

   See Also: "Using the Cluster Verification Utility" for more information about enabling and using the CVU

Note: For the reference node, select a node from your existing cluster nodes against which you want CVU to compare, for example, the newly added nodes that you specify with the `-n` option.

### Checking the Installation

To verify that your installation is configured correctly, perform the following steps:

1. Ensure that the new nodes can access the private interconnect. This interconnect must be properly configured before you can complete the procedures in "Step 2: Extending Clusterware and Oracle Software to New Nodes".

2. If you are not using a cluster file system, then determine the location on which your cluster software was installed on the existing nodes. Make sure that you have at least 250MB of free space on the same location on each of the new nodes to install Oracle Clusterware. In addition, ensure you have enough free space on each new node to install the Oracle binaries.

3. Ensure that the Oracle Cluster Registry (OCR) and the voting disk are accessible by the new nodes using the same path as the other nodes use. In addition, the OCR and voting disk devices must have the same permissions as on the existing nodes.

4. Verify user equivalence to and from an existing node to the new nodes using `rsh` or `ssh`.

After completing the procedures in this section, your new nodes are connected to the cluster and configured with the required software to make them visible to the clusterware. Configure the new nodes as members of the cluster by extending the cluster software to the new nodes as described in "Step 2: Extending Clusterware and Oracle Software to New Nodes".

Note: Do not change a hostname after the Oracle Clusterware installation. This includes adding or deleting a domain qualification.

## Step 2: Extending Clusterware and Oracle Software to New Nodes

The following topics describe how to add new nodes to the clusterware and to the Oracle database software layers using OUI:

- Adding Nodes at the Vendor Clusterware Layer

- Adding Nodes at the Oracle Clusterware Layer

### Adding Nodes at the Vendor Clusterware Layer

Add the new nodes at the clusterware layer according to the vendor clusterware documentation. For systems using shared storage for the Oracle Clusterware home, ensure that the existing clusterware is accessible by the new nodes. Also ensure that the new nodes can be brought online as part of the existing cluster. Proceed to the next section to add the nodes at the clusterware layer.

### Adding Nodes at the Oracle Clusterware Layer

Before beginning this procedure, ensure that your existing nodes have the `$CRS_HOME` environment variable set correctly. The OUI requires access to the private interconnect that you verified as part of the installation validation in Step 1. If OUI cannot make the required connections, then you will not be able to complete the following steps to add nodes.

Note: Instead of performing the first six steps of this procedure, you can alternatively run the `addNode.sh` script in silent mode as described at the end of this section.

1. On one of the existing nodes go to the `CRS_home``/oui/bin` directory and run the `addNode.sh` script to start OUI.

2. The OUI runs in `add node` mode and the OUI Welcome page appears. Click Next and the Specify Cluster Nodes for Node Addition page appears.

3. If you are using vendor clusterware, then the public node names automatically appear in the lower table. Click Next and OUI verifies connectivity on the existing nodes and on the new nodes. The verifications that OUI performs include determining whether:

   - The nodes are up

   - The nodes and private node names are accessible by way of the network

     Note: If any of the existing nodes are down, you can proceed with the procedure. However, once the nodes are up, you must run the following command on each of those nodes:
         runInstaller -updateNodeList -local
         "CLUSTER_NODES={available_node_list}"
         ORACLE_HOME=CRS_home
     This operation should be run from the `CRS_home``/oui/bin` directory and the `available_node_list` values is comma-delimited list of all of nodes currently in the cluster and `CRS_home` defines the Oracle Clusterware home directory.

   - The virtual hostnames are not already in use on the network.

4. If any verifications fail, then OUI re-displays the Specify Cluster Nodes for Node Addition page with a Status column in both tables indicating errors. Correct the errors or deselect the nodes that have errors and proceed. However, you cannot deselect existing nodes; you must correct problems on nodes that are already part of your cluster before you can proceed with node addition. If all the checks succeed, then OUI displays the Node Addition Summary page.

   Note: Oracle strongly recommends that you install Oracle Clusterware on every node in the cluster on which you have installed vendor clusterware.

5. The Node Addition Summary page displays the following information showing the products that are installed in the Oracle Clusterware home that you are extending to the new nodes:

   - The source for the add node process, which in this case is the Oracle Clusterware home

   - The private node names that you entered for the new nodes

   - The new nodes that you entered

   - The required and available space on the new nodes

   - The installed products listing the products that are already installed on the existing Oracle Clusterware home

   Click Next and OUI displays the Cluster Node Addition Progress page.

6. The Cluster Node Addition Progress page shows the status of the cluster node addition process. The table on this page has two columns showing the four phases of the node addition process and the phases' statuses as follows:

   - Instantiate Root Scripts—Instantiates `rootaddNode.sh` with the public nodes, private node names, and virtual hostnames that you entered on the Cluster Node Addition page.

   - Copy the Oracle Clusterware home to the New Nodes—Copies the Oracle Clusterware home to the new nodes unless the Oracle Clusterware home is on a cluster file system.

   - Save Cluster Inventory—Updates the node list associated with the Oracle Clusterware home and its inventory.

   - Run `rootaddNode.sh` and `root.sh`—Displays a dialog prompting you to run the `rootaddNode.sh` script from the local node (the node on which you are running OUI) and to run the `root.sh` script on the new nodes. If OUI detects that the new nodes do not have an inventory location, then OUI instructs you to run `orainstRoot.sh` on those nodes. The central inventory location is the same as that of the local node. The `addNodeActions` `timestamp``.log` file, where `timestamp` shows the session start date and time, contains information about which scripts you need to run and on which nodes you need to run them.

   The Cluster Node Addition Progress page's Status column displays `In Progress` while the phase is in progress, `Suspended` when the phase is pending execution, and `Succeeded` after the phase completes. On completion, click Exit to end the OUI session. After OUI displays the End of Node Addition page, click Exit to end the OUI session.

7. Obtain the remote port number, which you will use in the next step, using the following command from the `CRS_home``/opmn/conf` directory:

       cat ons.config

8. Run the `racgons` utility from the `bin` subdirectory of the Oracle Clusterware home to configure the Oracle Notification Services (ONS) port number. Use the following command, supplying the name of the node that you are adding for `new_node_name` and the remote port number obtained in the previous step:

       racgons add_config new_node_name:remote_port

9. Check that your cluster is integrated and that the cluster is not divided into partitions by completing the following operations:

   - Run the following CVU command to obtain detailed output for verifying cluster manager integrity on all of the nodes that are part of your RAC environment:

         cluvfy comp clumgr -n all [-verbose]

   - Use the following CVU command to obtain detailed output for verifying cluster integrity on all of the nodes that are part of your RAC environment:

         cluvfy comp clu [-verbose]

   - Use the following command to perform an integrated validation of the Oracle Clusterware setup on all of the configured nodes, both the pre-existing nodes and the nodes that you have added:

         cluvfy comp stage -post crinst -n all [-verbose]

   See Also: "Using the Cluster Verification Utility" for more information about enabling and using the CVU

You can optionally run `addNode.sh` in silent mode, replacing steps 1 through 7, as follows where `nodeI`, `nodeI+1`, and so on are the new nodes that you are adding:

    addNode.sh -silent "CLUSTER_NEW_NODES={nodeI, nodeI+1, … nodeI+n}"
    "CLUSTER_NEW_PRIVATE_NODE_NAMES={pnI, pnI+1, … pnI+n}"
    "CLUSTER_NEW_VIRTUAL_HOSTNAMES={vipI, vipI+1,…,vipI+n}"

You can alternatively specify the `variable=value` entries in a response file and run the `addNode` script as follows:

    addNode.sh -silent -responseFile filename OR addNode.bat -silent -responseFile filename

See Also: Oracle Universal Installer and OPatch User's Guide for details about how to configure command-line response files

Note: command-line values always override response file values.

Run `rootaddNode.sh` on the local node, or the node on which you are performing this procedure, and run `root.sh` on the new nodes. If OUI detects that the new nodes do not have an inventory location, then OUI instructs you to run `orainstRoot.sh` on those nodes. The central inventory location is the same as that of the local node. The `addNodeActions` `timestamp``.log` file, where `timestamp` shows the session start date and time, contains the information about which scripts you need to run and on which nodes you need to run them.

After you have completed the procedures in this section for adding nodes at the Oracle Clusterware layer, you have successfully extended the Oracle Clusterware home from your existing the Oracle Clusterware home to the new nodes. Proceed to "Step 3: Preparing Storage on New Nodes" to prepare storage for RAC on the new nodes.

## Step 3: Preparing Storage on New Nodes

To extend an existing RAC database to your new nodes, configure the shared storage for the new instances to be added on new nodes so that the storage type is the same as the storage that is already used by the existing nodes' instances. Prepare the same type of storage on the new nodes as you are using on the other nodes in the RAC environment that you want to extend as follows:

- Automatic Storage Management (ASM)

  If you are using ASM, then make sure that the new nodes can access the ASM disks with the same permissions as the existing nodes.

- Oracle Cluster File System (OCFS)

  If you are using Oracle Cluster File Systems, then make sure that the new nodes can access the cluster file systems in the same way that the other nodes access them.

- Vendor Cluster File Systems

  If your cluster database uses vendor cluster file systems, then configure the new nodes to use the vendor cluster file systems. Refer to the vendor clusterware documentation for the pre-installation steps for your UNIX platform.

- Raw Device Storage

  If your cluster database uses raw devices, then prepare the new raw devices by following the procedures described in the next section.

See Also: Oracle Database Oracle Clusterware and Oracle Real Application Clusters Installation Guide for Microsoft Windows for more information about the Oracle Cluster File System

Run the following command to verify your cluster file system and obtain detailed output where `nodelist` includes both the pre-existing nodes and the newly added nodes and `file system` is the name of the file system that you used for the Oracle Cluster File System:

    cluvfy comp cfs -n nodelist -f file system [-verbose]

See Also: "Using the Cluster Verification Utility" for more information about enabling and using the CVU

### Raw Device Storage Preparation for New Nodes

To prepare raw device storage on the new nodes, you need at least two new disk partitions to accommodate the redo logs for each new instance. Make these disk partitions the same size as the redo log partitions that you configured for the existing nodes' instances. Also create an additional logical partition for the undo tablespace for automatic undo management.

On applicable operating systems, you can create symbolic links to your raw devices. Optionally, you can create a raw device mapping file and set the `DBCA_RAW_CONFIG` environment variable so that it points to the raw device mapping file. Use your vendor-supplied tools to configure the required raw storage.

See Also: Your platform-specific Real Application Clusters installation and configuration guide for procedures about using DBCA to create and delete RAC databases

Run the following command to verify that the prepared raw device storage is accessible from all of the configured cluster nodes where `node_list` includes both the pre-existing nodes and the newly added nodes and `storageID_list` is a comma-delimited list of storage identifiers:

    cluvfy comp ssa [ -n node_list ] [ -s storageID_list ] [-verbose]

See Also: "Using the Cluster Verification Utility" for more information about enabling and using the CVU

After completing the procedures in this section, you have configured your shared storage for use by the instances to be added on the new nodes so that the new nodes can access the Oracle software. Additionally, the existing nodes can access the new nodes and instances. Use OUI as described in the procedures in "Step 4: Adding Nodes at the Oracle RAC Database Layer" to configure the new nodes at the RAC database layer.

## Step 4: Adding Nodes at the Oracle RAC Database Layer

To add nodes at the Oracle RAC database later, run OUI in `add` `node` mode to configure your new nodes. Before beginning this procedure, ensure that your existing nodes have the `$ORACLE_HOME` environment variable set correctly. If you have multiple Oracle homes, then perform the following steps for each Oracle home that you want to include on the new nodes:

Note: Instead of performing the first six steps of this procedure, you can alternatively run `addNode.sh` in silent mode as described at the end of this section.

1. On an existing node from the `Oracle_home``/oui/bin` directory, run the `addNode.sh` script. This starts OUI in `add` `node` mode and displays the OUI Welcome page. Click Next on the Welcome page and OUI displays the Specify Cluster Nodes for Node Addition page.

2. The OUI verifies connectivity and performs availability checks on the existing nodes and on the nodes that you want to add. Some of checks performed determine whether:

3. - The nodes are up

   - The nodes are accessible by way of the network

4. The Node Addition Summary page has the following information about the products that are installed in the Oracle home that you are going to extend to the new nodes:

   - The source for the add node process, which in this case is the Oracle home

   - The existing nodes and new nodes

   - The new nodes that you selected

   - The required and available space on the new nodes

   - The installed products listing all of the products that are already installed in the existing Oracle home

   Click Finish and OUI displays the Cluster Node Addition Progress page.

5. The Cluster Node Addition Progress page shows the status of the cluster node addition process. The table on this page has two columns showing the four phases of the node addition process and each phases' status as follows:

   - Copy the Oracle Home to the New Nodes—Copies the entire Oracle home from the local node to the new nodes unless the Oracle home is on a cluster file system

   - Save Cluster Inventory—Updates the node list associated with the Oracle home and its inventory

   - Run `root.sh`—Displays the dialog prompting you to run `root.sh` on the new nodes

   The Cluster Node Addition Progress page's Status column displays `Succeeded` if the phase completes, `In` `Progress` if the phase is in progress, and `Suspended` when the phase is pending execution. After OUI displays the End of Node Addition page, click Exit to end the OUI session.

6. Run the `root.sh` script on all of the new nodes.

7. Add a Listener to the new node by running the Net Configuration Assistant (NetCA) from new node and selecting only the new node on the Node Selection page.

You can optionally run `addNode.sh` in silent mode, replacing steps 1 through 6, as follows where `nodeI`, `nodeI+1`, and so on are the new nodes that you are adding:

    addNode.sh -silent "CLUSTER_NEW_NODES={nodeI, nodeI+1, … nodeI+n}"

You can also specify the `variable=value` entries in a response file, known as `filename`, and you can run the `addNode` script as follows:

    addNode.sh -silent -responseFile filename

Command-line values always override response file values.

See Also: Oracle Universal Installer and OPatch User's Guide for more information about how to configure command-line response files and Oracle Database Net Services Administrator's Guide for more information about NETCA

After completing the procedures in this section, you have defined the new nodes at the cluster database layer. You can now add database instances to the new nodes as described in "Step 5: Adding Database Instances to New Nodes".

## Step 5: Adding Database Instances to New Nodes

You can use Enterprise Manager, or the Database Configuration Assistant (DBCA) in either interactive mode or in silent mode, to add database instances to new nodes. Before beginning this procedure, ensure that your existing nodes have the `$ORACLE_HOME` environment variable set correctly. Use one of the following procedures to add the database instances to the new nodes:

- Using Enterprise Manager to Add Database Instances to New Nodes

- Using DBCA in Interactive Mode to Add Database Instances to New Nodes

- Using DBCA in Silent Mode to Add Database Instances to New Nodes

### Using Enterprise Manager to Add Database Instances to New Nodes

To add a database instance to a new node with Enterprise Manager, perform the following procedure:

1. From the Cluster Database Home page, click the Maintenance tab.

2. Under the Deployments section, click Add Instance. This initiates a wizard to guide you through the instance addition process.

3. Perform the following tasks for each wizard step:

   - If the database uses ASM, then provide host and ASM credentials.

   - Specify the hosts for which you want to add instances.

   - Review and submit the job.

   After you submit the job, the wizard provides a summary page that shows whether the job succeeded and the wizard also provides other detailed information about the job, such as the elapsed time and an output log.

After adding the instances to the new nodes using the steps described in this section, perform any needed service configuration procedures as described in Chapter 6, "Introduction to Workload Management".

### Using DBCA in Interactive Mode to Add Database Instances to New Nodes

To add a database instance to a new node with DBCA in interactive mode, perform the following procedure:

1. Start the Database Configuration Assistant (DBCA) by entering `dbca` at the system prompt from the `bin` directory in the `Oracle_home` directory.

   The DBCA displays the Welcome page for RAC. Click Help on any DBCA page for additional information.

2. Select Real Application Clusters database, click Next, and DBCA displays the Operations page.

3. Select Instance Management, click Next, and DBCA displays the Instance Management page.

4. Select Add Instance and click Next. The DBCA displays the List of Cluster Databases page that shows the databases and their current status, such as `ACTIVE`, or `INACTIVE`.

5. From the List of Cluster Databases page, select the active RAC database to which you want to add an instance. Enter user name and password for the database user that has SYSDBA privileges. Click Next and DBCA displays the List of Cluster Database Instances page showing the names of the existing instances for the RAC database that you selected.

6. Click Next to add a new instance and DBCA displays the Adding an Instance page.

7. On the Adding an Instance page, enter the instance name in the field at the top of this page if the instance name that DBCA provides does not match your existing instance naming scheme. Then select the new node name from the list, click Next, and DBCA displays the Services Page.

8. Enter the services information for the new node's instance, click Next, and DBCA displays the Instance Storage page.

9. If you are using raw devices or raw partitions, then on the Instance Storage page select the Tablespaces folder and expand it. Select the undo tablespace storage object and a dialog appears on the right-hand side. Change the default datafile name to the raw device name for the tablespace.

10. If you are using raw devices or raw partitions or if you want to change the default redo log group file name, then on the Instance Storage page select and expand the Redo Log Groups folder. For each redo log group number that you select, DBCA displays another dialog box. Enter the raw device name that you created in the section "Raw Device Storage Preparation for New Nodes" in the File Name field.

11. If you are using a cluster file system, then click Finish on the Instance Storage page. If you are using raw devices, then repeat step 10 for all of the other redo log groups, click Finish, and DBCA displays a Summary dialog.

12. Review the information on the Summary dialog and click OK or click Cancel to end the instance addition operation. The DBCA displays a progress dialog showing DBCA performing the instance addition operation. When DBCA completes the instance addition operation, DBCA displays a dialog asking whether you want to perform another operation.

13. After you terminate your DBCA session, run the following command to verify the administrative privileges on the new node and obtain detailed information about these privileges where nodelist consists of the newly added nodes:

        cluvfy comp admprv -o db_config -d oracle_home -n nodelist [-verbose]

After adding the instances to the new nodes using the steps described in this section, perform any needed service configuration procedures as described in Chapter 6, "Introduction to Workload Management".

### Using DBCA in Silent Mode to Add Database Instances to New Nodes

You can use the Database Configuration Assistant (DBCA) in silent mode to add instances to nodes onto which you have extended an Oracle Clusterware home and an Oracle home. Use the following syntax to perform this operation where `node` is the node onto which you want to add the instance, `gdbname` is the global database name, `instname` is the name of the new instance, `sysdba` is the name of an Oracle user with SYSDBA privileges, and `password` is the password for the user name in `sysdba`:

    dbca -silent -addInstance -nodeList node -gdbName gdbname [-instanceName instname] -sysDBAUserName sysdba -sysDBAPassword password

Note that you only need to provide an instance name if you want to override the Oracle naming convention for RAC instance names.

After you have completed either of the DBCA procedures in this section, DBCA has successfully added the new instance to the new node and completed the following steps:

- Created and started an ASM instance on each new node if the existing instances were using ASM

- Created a new database instance on each new node

- Created and configured high availability components

- Created the Oracle Net configuration

- Started the new instance

- Created and started services if you entered services information on the Services Configuration page

After adding the instances to the new nodes using the steps described in this section, perform any needed service configuration procedures as described in Chapter 6, "Introduction to Workload Management".

## Adding Nodes that Already Have Clusterware and Oracle Software to a Cluster

Before beginning this procedure, ensure that your existing nodes have the `$CRS_HOME` and `$ORACLE_HOME` environment variables set correctly. To add nodes to a cluster that already have clusterware and Oracle software installed on them, you must configure the new nodes with the Oracle software that is on the existing nodes of the cluster. To do this, you must run two versions of an OUI process: one for the clusterware and one for the database layer as described in the following procedures:

1. Add new nodes at the Oracle Clusterware layer by running OUI from the Oracle Clusterware home on an existing node, using the following command:

       CRS_home/oui/bin/addNode.sh -noCopy

2. Add new nodes at the Oracle software layer by running OUI from the Oracle home as follows:

       Oracle_home/oui/bin/addNode.sh -noCopy

In the `-noCopy` mode, OUI performs all add node operations except for the copying of software to the new nodes.

Note: Oracle recommends that you back up your voting disk and OCR files after you complete the node addition process.

## Overview of Node Deletion Procedures

This section explains how to delete nodes from clusters using detailed manual procedures. The details of these steps appear in the following sections:

- Step 1: Deleting Instances from Real Application Clusters Databases

- Step 2: Deleting Nodes from Real Application Clusters Databases

- Step 3: ASM Instance Clean-Up Procedures for Node Deletion

## Step 1: Deleting Instances from Real Application Clusters Databases

The procedures in this section explain how to use Enterprise Manager, or DBCA in interactive or silent mode, to delete an instance from a RAC database.

- Using Enterprise Manager to Delete Database Instances from Existing Nodes

- Using DBCA in Interactive Mode to Delete Database Instances from Existing Nodes

- Using DBCA in Silent Mode to Delete Instance from Existing Nodes

### Using Enterprise Manager to Delete Database Instances from Existing Nodes

To delete an instance with Enterprise Manager from an existing node, perform the following steps:

1. From the Cluster Database Home page, click the Maintenance tab.

2. Under the Deployments section, click Delete Instance. This action initiates a wizard to guide you through the deletion process.

### Using DBCA in Interactive Mode to Delete Database Instances from Existing Nodes

To delete an instance using DBCA in interactive mode, perform the following steps:

1. Start DBCA on a node other than the node that hosts the instance that you want to delete. On the DBCA Welcome page select Oracle Real Application Clusters Database, click Next, and DBCA displays the Operations page.

2. On the DBCA Operations page, select Instance Management, click Next, and DBCA displays the Instance Management page.

3. On the Instance Management page, Select Delete Instance, click Next, and DBCA displays the List of Cluster Databases page.

4. Select a RAC database from which to delete an instance. Enter a user name and password for the database user that has SYSDBA privileges. Click Next and DBCA displays the List of Cluster Database Instances page. The List of Cluster Database Instances page shows the instances that are associated with the RAC database that you selected and the status of each instance.

5. Select an instance to delete and click Finish.

6. If you have services assigned to this instance, then the DBCA Services Management page appears. Use this feature to reassign services from this instance to other instances in the cluster database.

7. Review the information about the instance deletion operation on the Summary page and click OK. Otherwise, click Cancel to cancel the instance deletion operation. If you click OK, then DBCA displays a Confirmation dialog.

8. Click OK on the Confirmation dialog to proceed with the instance deletion operation and DBCA displays a progress dialog showing that DBCA is performing the instance deletion operation. During this operation, DBCA removes the instance and the instance's Oracle Net configuration. When DBCA completes this operation, DBCA displays a dialog asking whether you want to perform another operation.

9. Click No and exit DBCA or click Yes to perform another operation. If you click Yes, then DBCA displays the Operations page.

### Using DBCA in Silent Mode to Delete Instance from Existing Nodes

Use DBCA to delete a database instance from a node as follows, where the variables are the same as those in the preceding add instance command:

    dbca -silent -deleteInstance [-nodeList node] -gdbName gdbname -instanceName instname -sysDBAUserName sysdba -sysDBAPassword password

You only need to provide a node name if you are deleting an instance from a node other than the one on which you are running DBCA.

At this point, you have accomplished the following:

- De-registered the selected instance from its associated Oracle Net Services listeners

- Deleted the selected database instance from the instance's configured node

- Removed the Oracle Net configuration

- Deleted the Oracle Flexible Architecture directory structure from the instance's configured node.

## Step 2: Deleting Nodes from Real Application Clusters Databases

Before beginning these procedures, ensure that your existing nodes have the `$CRS_HOME` and `$ORACLE_HOME` environment variables set correctly. Use the following procedures to delete nodes from Oracle clusters on UNIX-based systems:

Note: You can perform some of the steps in this procedure in silent mode as described at the end of this section.

1. If there are instances on the node that you want to delete, then perform the procedures in the section titled "Step 1: Deleting Instances from Real Application Clusters Databases" before executing these procedures. If you are deleting more than one node, then delete the instances from all the nodes that you are going to delete.

2. If you use ASM, then perform the procedures in the following section, "Step 3: ASM Instance Clean-Up Procedures for Node Deletion".

3. If this is the Oracle home from which the node-specific listener named `LISTENER_``nodename` runs, then use NETCA to remove this listener. If necessary, re-create this listener in another home.

   See Also: Oracle Database Net Services Administrator's Guide for more information about NETCA

4. Depending on whether you have a shared or non-shared Oracle home, complete one of the following two procedures:

   - For a shared home, run the following command on each of the nodes that are to be deleted:

         runInstaller -detachHome -local ORACLE_HOME=Oracle_home

   - For a non-shared home, on each node that you are deleting, perform the following two steps:

     - Run the following command:

           runInstaller -updateNodeList ORACLE_HOME=Oracle_home
           CLUSTER_NODES="" –local

       The `runInstaller` command is located in the directory `Oracle_home``/oui/bin`. Using this command does not launch an installer GUI.

     - Run OUI from the home and deinstall this home. Make sure that you choose the home to be removed and not just the products under that home.

5. From an existing node, run the following command:

       runInstaller -updateNodeList ORACLE_HOME=Oracle_home "CLUSTER_NODES={nodelist}"

   where `nodelist` is a comma-delimited list of nodes that remain in the cluster.

6. On the node that you are deleting, run the command `CRS_home``/install/rootdelete.sh` to disable the Oracle Clusterware applications that are on the node. Only run this command once and use the `nosharedhome` argument if you are using a local file system. The default for this command is `sharedhome` which prevents you from updating the permissions of local files such that they can be removed by the `oracle` user.

   If the `ocr.loc` file is on a shared file system, then run the command `CRS_home``/install/rootdelete.sh remote sharedvar`. If the `ocr.loc` file is not on a shared file system, then run the `CRS_home``/install/rootdelete.sh remote nosharedvar` command.

   If you are deleting more than one node from your cluster, then repeat this step on each node that you are deleting.

7. Depending on whether you have a shared or non-shared Oracle Clusterware home, complete one of the following two procedures:

   - For a shared home, run the following command on each of the nodes that are to be deleted, providing the shared home location for `CRS_home:`

         runInstaller -detachHome -local ORACLE_HOME=CRS_home

   - For a non-shared home, on each node that you are deleting, perform the following two steps:

     - Run the following command:

           runInstaller -updateNodeList ORACLE_HOME=CRS_home
           CLUSTER_NODES="" –local CRS=true

       The `runInstaller` command is located in the directory `CRS_home``/oui/bin`. Executing this command does not launch an installer GUI.

     - Run OUI from the home and deinstall this home. Make sure that you choose the home to be removed and not just the products under that home.

8. From an existing node, run the following command:

       runInstaller -updateNodeList ORACLE_HOME=CRS_home
       "CLUSTER_NODES={nodelist}"

   where `nodelist` is a comma-delimited list of nodes that remain in the cluster.

9. Run the following command to verify that the node is no longer a member of the cluster and to verify that the Oracle Clusterware components have been removed from this node:

       cluvfy comp crs -n all [-verbose]

   The response from this command should not contain any information about the node that you deleted; the deleted node should no longer have the Oracle Clusterware components on it. This verifies that you have deleted the node from the cluster.

   See Also: "Using the Cluster Verification Utility" for more information about enabling and using the CVU

As mentioned earlier in this procedure, you can optionally delete nodes from Real Application Clusters databases in silent mode by completing the following steps:

1. Complete steps 1 through 3 of the procedure described at the start of this section under the heading "Step 2: Deleting Nodes from Real Application Clusters Databases" .

2. Depending on whether you have a shared or non-shared Oracle home, complete one of the following two procedures:

   - For a shared home, run the following command on each of the nodes that are to be deleted:

         ./runInstaller -detachHome -local ORACLE_HOME=Oracle_home

   - For a non-shared home, on each node that you are deleting, perform the following two steps:

     - Run the following command:

           runInstaller -updateNodeList ORACLE_HOME=Oracle_home
           CLUSTER_NODES="" –local

       The `runInstaller` command is located in the directory `Oracle_home``/oui/bin`. Using this command does not launch an installer GUI.

     - De-install the Oracle home from the node that you are deleting by running the following command from the `Oracle_home``/oui/bin` directory:

           ./runInstaller -deinstall -silent "REMOVE_HOMES={Oracle_home}"

3. Complete steps 5 through 8 from the procedure described at the start of this section that begins.

4. Depending on whether you have a shared or non-shared Oracle Clusterware home, complete one of the following two procedures:

   - For shared homes, do not perform a de-install operation. Instead, perform a detach home operation on the node that you are deleting. To do this, run the following command from `CRS_home``/oui/bin`:

         ./runInstaller -detachHome ORACLE_HOME=CRS_home

   - For a non-shared home, on each node that you are deleting, perform the following two steps:

     - Run the following command:

           runInstaller -updateNodeList ORACLE_HOME=CRS_home
           CLUSTER_NODES="" –local CRS=true

       The `runInstaller` command is located in the directory `CRS_home``/oui/bin`. Executing this command does not launch an installer GUI.

     - On each node that you are deleting, perform the following step from the `CRS_home``/oui/bin` directory:

           ./runInstaller -deinstall -silent "REMOVE_HOMES={CRS_home}"

       where `CRS_home` is the name given to the Oracle Clusterware home you are deleting.

5. Complete steps 10 and 11 of the procedure described at the start of this section.

## Step 3: ASM Instance Clean-Up Procedures for Node Deletion

If you are using ASM, then perform the following procedure to remove the ASM instances:

1. Stop all of the databases that use the ASM instance that is running from the Oracle home that is on the node that you are deleting.

2. If you are not using a cluster file system for your ASM Oracle home, then run the `rm` or `delete` commands mentioned in the previous step on each node on which the Oracle home exists.